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Abstract. We show that for any infinite set of unit vectors U in the 
maximal operator defined by 

Huf{x) = sup 
ueu 

is not bounded in L^(R^). 



1. Introduction 

For a rapidly decreasing function / and a unit vector u — {cos 6, sin 6'), 9 G [0, 2tt], 
we define 

r f(x-tcos0^-tsin^ 
Huf{x,y) =p.v. dt, 

J~oo t 

which is the one dimensional Hilbert transform along the direction u. It is well 
known this operator can be extended to a bounded operator from L^'(M^) to itself 
when 1 < p < oo. In this paper we study operators 

(1) Hu f (x, y) ^ sup \Huf{x,y)\, 

ueu 

where ?7 is a set of unit vectors u in R^. Analogous operators for the maximal 
functions are properly investigated. The case of lacunary U first are considered in 
the papers [TH], [S]- A final result is obtained by A. Nagel, E. M. Stein and 
S. Wainger in [151. They proved the boundedness of the norms of these operators 
in LP, 1 < p < oo, for a lacunary U. Upper bounds of such operators depending 
on the cardinahty #[/ of the set U considered in the papers [4] . [5] . [6] . [I3] . [I4] . [I9] . 
And the definitive estimates due to N. Katz ([13], [S]). In the papers he obtained 
a logarithmic order for the norms of two different maximal operators depending 
on #J7. Various generalizations of these results are considered in series of papers 
([1],[2],I3],[101,[171). 

As for the operator ([1]) there was no any results, but the bound 

\\Huf{x)\\L2<\0g#U\\fh2. 

This is an immediate consequence of Menshov-Rademacher theorem (see [3] or |12)). 
in spite of the fact that in Katz's theorems subtle range of ideas are used. It was 
not even known whether Hjj is bounded in or not for an infinite lacunary set 
U. The main result of the paper is unboundedness of Hjj in for any infinite U. 

Theorem 1. For any infinite set of unit vectors U the operator Hjj can not he 
extended to a bounded operator from Lp'{M?) to L^iM?) for all 1 < p < oo. 



f(x - tu) 
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This theorem is an immediate consequence of the following estimate. 



Theorem 2. If U is a finite set and 1 < p < oo then \\Hij\\]^2^i^p > cy/\og #J7, 
where c > is an absolute constant. 

The author is very grateful to M. T. Lacey for a nice hospitality during two visits 
at the Georgia Tech in 2003 and 2004, and for helpful conversations about these 
problems. 

2. Proof of Theorems 

Let fn{x), n = 1,2, ■ ■ ■ , 2™ — 1 (/„ ^ 0) be a system of functions defined on the 
square 

Q = [-TT, Tt] X [-TT, Tt]. 

In some places we shall use for fn{x) double numbering, defining by 

f^''Hx) = Ux), n = 2'=+j-l, 1<J<2^ k = 0,l,--- ,m-l. 
We shall say the sequence fn{x) = ff'\x) is tree-system, if 

(2) supp f J+i') C {x e Q : (x) > 0}, supp f C {x e Q : fj^"^^ (x) < 0}. 
Applying ^ several times we get 



supp fp+'^ C {x e Q : if"'^ (x) > 0} ^ i e (2'-j - 2\ 2'-j - 2'' 
supp ff C {x e Q : if"^ (x) < 0} ^ i e (2''j - 2'-i, 2''j] , 



and then 

(4) suppf/''+''^ n {x e Q : fj^''^(x) > 0} = ^ i^ (2'-j - 2\ 2'-j - 2'^"!] , 

(5) suppfp+'^^ n {x e Q : fj^''^(x) < 0} = ^ i ^ (2'-j - 2'-i, 2''j] . 

The following lemma for the Haar system is proved by E. M. Nikishin and P. L. Ul'yanov 
[IB] and we use the same idea to prove a general one (see also |12j). 

Lemma 3. // fn{x), x € Q, n = 1,2, ■ ■ ■ , 2™ — 1, is tree-system then there exists 
a permutation a of the numbers {1, 2, • • • , 2™ — 1} such that 

I ^ 2'"-l 

sup I V /ct(„) (a:) I > - V |/n(a;)|- 

l</<2'" 1 O , 

Proof. We connect with each fn{x), n = 2^^ + j — 1, a number 

tn = ^e[0,i]. 

Notice they are not equal for different n's. Define the permutation a so that 

t<7{l) < ta(2) < ■ ■ ■ < io-(2"'-l)- 

We shall prove that for any x E Q there exists a number I — l{x) with 

(6) fa(n){x) > 0, if n > l{x), 

(7) /,(„)(x) <0, if n</(x). 
Defining 

I = l{x) = sup{n : 1 < n < 2", /<,(„)(x) < 0}, 
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we shall have ([7]) immediately and \i v = a{l{x) + 1) then 
(8) Mx) > 0. 

To prove ([6]) it is enough to show that if dH) holds and tn > ti, then /„(a;) > . 
Suppose 

According to the assumption 

_2j -I 2i-l _ 
If s > A; then s = fc + 7- (r > 0) . From ^ we get 

Therefore by ([4]) we obtain 

suppfn n {f,. > 0} = 0. 

Since fu{x) > we get /„(a;) = 0. 

If s < A:, then k ^ s + r {r > 0). Applying Q we get 

Hence by ([5]) we have 

suppfi. n {fn < 0} 0. 
By dS]) we conclude fn{x) > 0. So ([H]) and ^ are proved. Using them, we obtain 



l<i<2 

If 



I l{x) 2"-l 

—1 n=l n—1 



then we get 



and therefore 



l(x) l(x) ^ 2"*-l 

n=l n—1 n—1 



2'^-! l{x) 
n=i(x) + l ri=l 



2'"-l 2"-! i(^) 

n=l n=i(a;) + l n=l 

^ 2'"-l 2™-l 

>2 X > 3 X 

n=Z(a;) + l n=l 

Thus we conclude 

I ^ 2"-l ^ 2"-! 

sup I X /T(n)(a;)| > o X = Q X 



l<i<2 



3 ^ ,.u^,.,v 3 

n—1 n—1 



□ 
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Fix a Schwartz function (f>{x) with 

(10) (j){x)>0, (t){x)dx^l, SUPP0 C [-1, 1]. 
We consider operators 

(11) <^>n{f) = ^n{x,yj)=n^ J f{x-t,y-s)(f>{nt)(l){ns)dtds, n = l,2,---. 
Applying HO]), for any / G L°°{M.'^) we get 

(12) inf /(x,y)<$„(a;, ?;,/)< sup f{x,y), 

(13) supp$„(f) C [-n,n] X [-n, n]. 

If in addition / is compactly supported, using a standard argument, we conclude 

(14) |l$„(/)-/|U2 ^0, asn^cx). 
If 

(15) n = 2'=+j-l, l<j<2^ A: = 0,1,--- ,m-l, 
then we denote 

n = 2^^-i + 



J + 1 



where [•] means an integer part of a number. Using this notation we may write the 
conditions ([2]) by 

(16) SUppfn C {(-l)j + l -ffi > 0}. 

We shall consider sectors defined by 

{{x,y) : x + iy = re"" , r > 0, a < 6* < /?} 

where Q < a < (3 < 2tt. Some arguments in the prove of following lemma are 
coming from the paper [llj . 

Lemma 4. Let Sn, n = 1, 2, • • • , i/ = 2"* — 1, be sectors on the plane. Then there 
exist functions fn G -L^(R^), n = 1, 2, • • • such that 

(17) suppf„cS„, 

(18) E;=ill/.lli.<ci, 

(19) \{{x,y) e Q : maxi<„<j. | Ej=i (a;, y)| > csVlogiy}] > C2, 
where a is the permutation from Lemma\^ and all the constants are absolute . 

Proof. We will assume (jisp everywhere below. For a given e > define sets En — 
Sf^ CQ {Ei^ Ef^ ^ Q), gn G L°°(IR2) and p„, (?„ £ Z with conditions 

a) En {{x,y) e En : (-1)^+^ cos(p,-ja; + qny) > O}, n = 2, 3, • • • , i/, 

b) < gn < 1, hn - Ib„||l2 < £, n = 1,2, • • • , j/, 

c) suppgn C S„ - (p„, q„), n = 1, 2, • • • ,1^, 

d) /^^ I cos(p„a; + qny)\dxdy > 

We do it by induction. Take Ei = e[^^ = Q. According to there exists I > 
with 

\\MIe,)-IeA\l- <e- 
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Define gi = ^i{Iei) and then applying p2|) we get b) for n=l. We note that if E 
is a measurable set then 



(20) / \ cos{px + qy)\dxdy > J cos {px + qy)dxdy 



This observation shows that for sufficiently large pi = p and qi — q we shall have 
condition d) for n = 1. On the other hand by (|13p suppgi is bounded. Thus for 
an appropriate pi, qi we will have also c) (with n = 1). Certainly we can choose pi 
and qi common for both conditions c) and d) . Now we suppose that the conditions 
a)-d) holds for any A: < n, in particular for h. We define En by the equality in a). 
Then we chose positive integer I with 

II^KIbJ-IbJIl^ <£. 

and denote gn — <&;(!£;„). Again applying ([20]) and using the boundedness of 
suppgn we may chose integers Pn,q_n satisfying c) and d) together. Using the 
condition a), it is easy to check that the sets £"„ satisfies the conditions 

Using this properties we conclude 

(21) I^^ (a;, y) = TO, almost everywhere on Q. 

n=l 

Now we define 

22 fn{x,y) = ^ . 

The condition p7|) immediately follows from c), because 

SUppfn = suppgn + (p„, qn)- 

On the other hand taking a small e by b) and ((2T|) we obtain 

— n TO — ' n mm ^ — ' n 

which gives (fTS]) . Now consider functions 

/oQ^ i sof Tt cos{p„x + qny)- gn{x,y) ■lEr.{x,y) 

Applying b) and (|^ we get 

(24) ||/„ - = I Wn? < f \fn? = - I \9n? < -■ 

On the other hand we have 

n n n u 

^^^^ i^„^<\rlE/-0)l > i^a^^lE^Z-O-)! ^ i^^ax^lE/-0)l -El/^- -^/j-l- 
j=i j=i j=i j=i 
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From (l24l) we obtain 



< 



ve 



L2 



m 



Therefore taking a small e > we can say that 
(26) 



<'5, 



for any given 5 > Q. From ([^5]) and we conclude, that to prove and so the 
lemma it is enough to prove 



(27) 



\{{x,y) e Q : max | V y)| > csy^logv}] > C2 



Let us show that /„ is a tree-system, i.e. it satisfies ([T6| . Since g„ > from ((23l) 
we get that fn{x,y) and cos(p„a; + qnV) have same sign in the set £"„. Therefore 
by a) we obtain 

suppfn C E„ = {(x, y) e Efi : (-l)j+^ cos(pnX + q^y) > O} 
= {{x,y) e En : {-iy+'fnix,y) > O} - {(x^y) e Er, : {-iy+'U{x,y) > O} 

= {(a^,y)eQ: (-l)^+74a:,y) >0}. 
Hence fn is tree-system. So according to Lemma [3] we have 

n 1 ^ 

(28) i^^^^lE'^'^O')^^'^) - 3 E 

From and the conditions b) and d) we get 



1 



1 



|/„| ^= / |5r„ cos(p„a; + qny)\dxdy > —j= \ \ cos(p„x + qny)\dxdy 



m Je. 



1(1 - 9n) cos{pnX + qny)\dxdy > 



\E„ 



If wc take e > to be small then from (PT|) we obtain 

•"•i j=l ^ n=l V 

Combining this and (^5]) we get 



(29) 



« n 



m. 



On the other hand by ([2T|) . ([22]) and b) for any (x, y) £ Q we have 

n ^ u 

^fa{3){x,y)\ < E l/i(2;,y)l < 7^ E (^' ^) - ^ 

J=l J=l J=l 



(30) max 

l<n<Ar 



From (Hi) and (EOl) follows (HJ 



□ 
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Proof of Theorem [H For any region S* C we denote 



Since the multiplier for the Hilbert transform is i ■ signx, for any direction u = 
(cos 0, sin 9) we have 

Hufix,y)^i- sign(xcos6' + ysin6')?(x,y). 

Thus we conclude 

(31) H^f = i{2-TrJ - f) 

where 

r„ = {(x, y) € : a; cos 6* + y sin 6* > 0}. 

Denote 

Ty/ = sup|rr„/|. 

Let U = {uk = (cosflfc, sinflfc) : fc = 1,2,-- - , A^} be the set of directions from 
Theorem 2. Without loss of generality we can assume 6^ G (0, 7r/2), di < 02 < 
■ ■ ■ < 9pf and N — 2™. According to |3T|) . to prove Theorem 2 it is enough to prove 
that 



(32) WTufh^ > Vl^ll/IU^ 

for some function / G L^{R'^). We denote by 5*^, fc = 1, 2, • • • , i/ = 2" - 1, 
the sectors obtained by the vectors (uk)'^ = (cos^fc, — sin^fc) and (ufe+i)^ — 
{cos9k+i,-smdk+i), i.e. 

(33) Sk = {{x,y) : x > 0,xcos9k + ysm6k > 0, xcos6k+i +ysin9k+i < 0}. 
Hence if we suppose 

suppf c y Sk 

k=l 

then we can write 



(34) Tuf{x,y)= sup 



'^TsJ{x,y) 



fc=i 



We define functions /„ satisfying the conditions of the Lemma [4] corresponding to 
the sectors S',^ = S„-i(^n) in (ESI)- Denote 

fc=i 

Since Sn are mutually disjoint the functions /„ are orthogonal. Thus by ITSl) 
we get ||/||l2 < c\. From p7|) we have suppfn C So.-i(n), n — 1,2,- •• i.e. 
supp fo.(n) C Sn and therefore 

TsJ{x) = /„(a;). 

According to ([M)) we obtain 

/ 

Tufix,y) = maji^\J2f<TU){x,y)\. 
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Using (UHl) and (HH), we get 

\{{x,y) e Q : Tuf{x,y) > C3^/k^}\ > 
And therefore \\Tuf\\Lp > VW^WfU^. □ 
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